           Towards a declarative, reproducible, homogeneous,

                 cross-platform command-line environment across

                                            remote HPC machines

                                  (with live demonstration pretending to be a masterhacker!)

                                             The STEP-UP RSLondon Conference 2025

                                  Dr Krishnakumar Gopalakrishnan, University College London

                                                        July 07, 2025

                                                      Who am I?

                                                    •  󱩊 Research Software Engineer at UCL ARC

                                                    •  󱖴 Rides bikes on weekends

                                                    •  󱖴 Uses Arch btw

                                                    •  󰵭 Test cricket fan

                                                                                                                       2 / 38

                                                      Who am I?

                                                    •  󱩊 Research Software Engineer at UCL ARC

                                                    •  󱖴 Rides bikes on weekends

                                                    •  󱖴 Uses Arch btw

                                                    •  󰵭 Test cricket fan

                                                    •   Command-line enthusiast

                                                           # So in the ongoing Anderson-Tendulkar Trophy

                                                         ——————————————————— [finished] ———————————————————

                                                                                                                       3 / 38

                                            The Unix Philosophy

           ▍ This is the Unix philosophy:

           ▍ Write programs that do one thing and do it well.

           ▍ Write programs to work together.

           ▍ Write programs to handle text streams, because that is a universal interface.

                                                         Doug McIlroy, Bell Labs Computing Sciences Research Center

           https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well

                                                                                                                       4 / 38

                        Command-line interfaces and HPC systems

              •  The default user interface on HPC systems

              •  Users are provided with Secure Shell (SSH) access

              •  Edit/Compile/Debug cycle on remote systems

              •  Job submission workflow and non-interactive compute node use that uses schedulers

                                                                                                                       5 / 38

                        Command-line interfaces and HPC systems

              •  The default user interface on HPC systems

              •  Users are provided with Secure Shell (SSH) access

              •  Edit/Compile/Debug cycle on remote systems

              •  Job submission workflow and non-interactive compute node use that uses schedulers

           ▍ 󰋽 Note

▍

           ▍ Often not emphasised: interactive use e.g. text editing, searching, command history ...

                                                                                                                       6 / 38

                              Towards a homogeneous environment

           ▍  Goal

▍

           ▍ Need an environment identical to personal workstation on all remote hosts (HPCs, VMs, Containers ...)

           ██ Workstation platforms (OS/CPU Architectures)

           Operating System │ x86 (32-bit) │ x86_64 (64-bit) │ ARMv7 (32-bit) │ ARM64 (AArch64) │ RISC-V │ PowerPC

           ─────────────────┼──────────────┼─────────────────┼────────────────┼─────────────────┼────────┼────────

           Windows          │              │                 │                │                 │        │

           Linux            │              │                 │                │                 │        │

           *BSD             │              │                 │                │                 │        │

           macOS            │              │                 │                │                 │        │

           AIX              │              │                 │                │                 │        │

           Haiku            │              │                 │                │                 │        │

           ReactOS          │              │                 │                │                 │        │

           RedoxOS          │              │                 │                │                 │        │

                                                                                                                       7 / 38

           ██ Realistic support matrix for combined Desktop/HPC

                                     Operating System │ x86_64 (64-bit) │ ARM64 (AArch64)

                                     ─────────────────┼─────────────────┼────────────────

                                     Linux            │ ✅              │ ✅

                                     macOS            │ ❓              │ ✅

                                     Windows          │ ❓              │ ❓

           ██ Common environment?

              •  Shell startup files (aliases, environment variables, cleanup ...)

              •  Cross-platform CLI/TUI tools

              •  Configuration files for tooling

                                                                                                                       8 / 38

                                               Hands-on example

                                                                                                                       9 / 38

                                                       Scenario

           ██ All hosts in the inventory

                                                                                                                      10 / 38

              Consider only remote (HPC) hosts in the inventory

                                        hostname

                                      ——————————————————— [finished] ———————————————————

                                        blend

                                                                                                                      11 / 38

           ██ Remote hosts in the inventory

                                                                                                                      12 / 38

                       Config file synchronisation across hosts

              •   Clone/sync on remote host with a Version Control System (VCS)

              •  A dotfiles manager (https://dotfiles.github.io/)

                 ◦  GNU Stow

                 ◦  dotbot

                 ◦  rcm

                 ◦  and many many more!

                                                                                                                      13 / 38

                                                        Chezmoi

              •  I have settled on using Chezmoi (pronounced shay-mwa)

              •  Full power of go templates

                                        # chezmoi data

                                      ——————————————————— [finished] ———————————————————

                                                                                                                      14 / 38

                                             Chezmoi templating

                                      $HOME/.config/git/config

                                      [user]

                                          {{- if eq chezmoi.hostname "blend" }}

                                          email = "personal_email@personal_host.co.uk"

                                          {{- else }}

                                          email = "work_email@work.ac.uk"

                                          {{- end }}

                                        user@blend $ chezmoi apply

                                      [user]

                                          email = "personal_email@personal_host.co.uk"                                15 / 38

                                          Deploy on all systems

           ██ Use a configuration management tool

              •   Ansible (agentless, push model) 

              •   Puppet

              •  󰭼 Chef

              •  Salt stack

              •  Pyinfra

                                                                                                                      16 / 38

                                               Tooling/Packages

                                                                                                                      17 / 38

                            Combinatorial explosion of packages

           ▍ Consider

           ▍ m: different processor architectures (target CPUs),

           ▍ n: different operating systems,

           ▍ p: different executable tools.

           Total number of binaries to be managed:

           Even with 3 CPU architectures and 20 tools to be managed, we have about 60 executables to manage, from

           which the valid binaries for each platform must be identified and deployed on the p-th machine.

                                                                                                                      18 / 38

                                     Managing package lifecycle

           ▓▓▓ Install (?), Update to latest, Pin versions of selected packages/tools, Remove packages

                                                                                                                      19 / 38

                                     Managing package lifecycle

           ▓▓▓ Install (?), Update to latest, Pin versions of selected packages/tools, Remove packages

             act                           cargo:tlrc                fx                        jinja-lsp

             actionlint                    ccache                    fzf                       jless

             age                           checkmake                 gdu                       jnv

             asdf:richin13/asdf            chezmoi                   gh                        jq

             -neovim                       clang-tools               git                       k9s

             awk-language-serve            cmake                     git-cliff                 kubectl

             r                             cosign                    gitui                     lazydocker

             bat                           cppcheck                  glow                      lazygit

             bat-extras                    croc                      go                        lazyjournal

             beautysh                      curl                      go:crictty                less

             bingrep                       delta                     go:daylight               ltex-ls

             binsider                      difftastic                go:kubecolor              lua

             biome                         direnv                    go:stew                   lua-language-serve

             bottom                        diskonaut                 goawk                     r                      20 / 38

             broot                         dive                      graphviz                  lychee

           Linters, Formatters, Language Runtimes, Compilation

                                      frameworks, Utilities ...

             minikube                      pipx                      procs                     usage

             nasm                          pipx:ansible-d            pylyzer                   uv

             neocmakelsp                   ev-tools                  python                    vault

             ninja                         pipx:ansible-i            ripgrep                   vlang

             node                          nventory-graph            ripgrep-all               viddy

             npm:ansible-langua            er                        ruff                      xz

             ge-server                     pipx:ansible-l            rust                      yamlfmt

             npm:mermaid-cli               int                       selene                    yamllint

             npm:compose-langua            pipx:argcomple            shellcheck                yj

             ge-service                    te                        shellharden               zig

             npm:alex                      pipx:bandit               shfmt                     zk

             npm:bash-language-            pipx:basedpyri            starship

             server                        ght                       stylua

             npm:bibtex-tidy               pipx:clang-for            taplo                                            21 / 38

             npm:commitlint                mat                       terraform                 # Let's count the

           How to manage these software tools/packages

                                                    everywhere?

                                                                                                                      22 / 38

           ██ System package manager?

              •  apt-get/apt

              •  yum/dnf

              •  zypper

              •  pacman

           ▓▓▓ Let's try

                                        which vim

                                      ————————————— [finished with error] ——————————————

                      which: no vim in

                      (/home/chezmoi_trial1/.config/pixi/condabin:/home/chezmoi_trial1/.local/bin:/home/

                      chezmoi_trial1/.local/share/mise/installs/act/0.2.79:/home/chezmoi_trial1/.local/s

                      hare/mise/installs/actionlint/1.7.7:/home/chezmoi_trial1/.local/share/mise/install

                      s/age/1.2.1/age:/home/chezmoi_trial1/.local/share/mise/installs/bat/0.25.0:/home/c

                      hezmoi_trial1/.local/share/mise/installs/bat-extras/2024.08.24/bin:/home/chezmoi_t              23 / 38

                      rial1/.local/share/mise/installs/biome/2.0.6:/home/chezmoi_trial1/.local/share/mis

                                             Let us install vim

                                        # which OS are we on?

                                      ——————————————————— [finished] ———————————————————

                                        NAME="blendOS"

                                        PRETTY_NAME="blendOS"

                                        ID=blendos

                                        BUILD_ID=rolling

                                        ANSI_COLOR="38;2;23;147;209"

                                        HOME_URL="https://blendos.co/"

                                        DOCUMENTATION_URL="https://docs.blendos.co/"

                                        SUPPORT_URL="https://github.com/blend-os"

                                        BUG_REPORT_URL="https://github.com/blend-os"

                                        LOGO=blendos-logo                                                             24 / 38

                                        pacman -S vim

                                      ————————————— [finished with error] ——————————————

                                  error: you cannot perform this operation unless you are

                                  root.

                                                                                                                      25 / 38

           ▓▓▓ Cannot do this on remote HPC systems. But let us on our local machine

                                        su krishnakumar

                                        sudo pacman -S vim

                                                                                                                      26 / 38

                    Software Package Management on a HPC system

              •  Curated list of scientific software pre-installed (and upon request)

              •  Versions are user-switchable by a modules system (environment-modules/Lmod)

              •  Custom user-modules can be loaded

              •  Unpriveleged container execution with container runtimes like Apptainer/Singularity/Podman?

                                                                                                                      27 / 38

                    Software Package Management on a HPC system

              •  Curated list of scientific software pre-installed (and upon request)

              •  Versions are user-switchable by a modules system (environment-modules/Lmod)

              •  Custom user-modules can be loaded

              •  Unpriveleged container execution with container runtimes like Apptainer/Singularity/Podman?

           ▍ 󰳦 🤨

▍

           ▍ But we are talking about installing simple productivity tools and keeping them up-to-date

           ▍ Not scientific libraries and programs

                                                                                                                      28 / 38

                                Popular HPC packaging ecosystem

                                                                                                                      29 / 38

              •  Intended for scientific software and libraries (GROMACS, LAMMPS, MPI ...)

              •  Support complex build provenances (MPI libraries, dependency versions, compilation toolchain ...)

              •  Latest versions of utility tools often not available

                                                                                                                      30 / 38

                            So, what are the options available?

                                                                                                                      31 / 38

                                             Nix was it for me!

                                                                                                                      32 / 38

                                                       However!

                                                                                                                      33 / 38

                                       So, is there a solution?

           ██ Desired characteristics

              •  Declarative/Idempotent

              •  Reproducible

              •  Installable/Runnable from user home directory

              •  Cross-platform

                                                                                                                      34 / 38

           ██ pixi global

              •  install

              •  edit

              •  sync

              •  manifest file pixi-global.toml

                                                                                                                      35 / 38

                                                                   ██ Mise

           ██ pixi global

              •  install                                              •  use -g

              •  edit                                                 •  prune

              •  sync                                                 •  global config.toml

              •  manifest file pixi-global.toml

                                                                                                                      36 / 38

                                       Keeping tools up to date

                                                                                                                      37 / 38

                                              Further resources

           ▓▓▓ Worflow configuration codes, presentation in HTML & PDF formats

                                                                                                                      38 / 38